Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(backend): add NoopTelemetryServiceImpl to make tel optional #2991

Merged
merged 5 commits into from
Oct 1, 2024

Conversation

BlairCurrey
Copy link
Contributor

@BlairCurrey BlairCurrey commented Sep 19, 2024

Changes proposed in this pull request

  • makes the telemetry service (not the actual collection) required and removes all the if(telemetry) checks
  • changes otel collector log config to get rid of a lot of noise that was making logs unreadable

Context

fixes #2980

Checklist

  • Related issues linked using fixes #number
  • Tests added/updated
  • Make sure that all checks pass
  • Bruno collection updated (if necessary)
  • Documentation issue created with user-docs label (if necessary)
  • OpenAPI specs updated (if necessary)

@github-actions github-actions bot added type: tests Testing related pkg: backend Changes in the backend package. type: source Changes business logic labels Sep 19, 2024
Copy link

netlify bot commented Sep 19, 2024

Deploy Preview for brilliant-pasca-3e80ec ready!

Name Link
🔨 Latest commit c8753fc
🔍 Latest deploy log https://app.netlify.com/sites/brilliant-pasca-3e80ec/deploys/66f304456430020008272ae1
😎 Deploy Preview https://deploy-preview-2991--brilliant-pasca-3e80ec.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lots of changed lines but its mostly just adding a level of nesting - I didnt change any existing tests. just added the disabled describe block, which ensures the correct telemetry service implementation is used (real one or noop) according to whether telemetry is enabled.

Comment on lines 48 to 55
export function createTelemetryService(
deps: TelemetryServiceDependencies
): TelemetryService {
if (!deps.enableTelemetry) {
return new NoopTelemetryServiceImpl(deps)
}
return new TelemetryServiceImpl(deps)
}
Copy link
Contributor Author

@BlairCurrey BlairCurrey Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel like I need to elaborate on this approach because it might deviate from the expectations in the issue a bit which said:

In order to still respect the ENABLE_TELEMETRY and ENABLE_TELEMETRY_TRACES flags, all we can do is just to not register metric/trace providers in the telemetry initialization.

What we're doing here is adding a new implementation that just doesnt do anything and using that if telemetry is disabled.

I can revise this if we don't like it but I think its working well.

I didn't do it as described for a couple reasons: 1) I don't think its possible - we arent registering metric/trace providers on service init. Traces are already registered outside the service so they seem to be a non-factor here. 2) I think even if we did we'd find ourselves in a scenario where we're calling methods like incrementCounterWithTransactionAmount, doing all that work (including network call) and then simply do nothing.

Alternatives considerd:

  1. in the spirit of the original suggestion, we could simply gate the counter create in getOrCreateCounter (and histogram) behind if telemetry is enabled and returned undefined if disabled like:
  private getOrCreateCounter(name: string, options?: MetricOptions): Counter {
    let counter = this.counters.get(name)
    if (!counter || !deps.enableTelemetry) {
      counter = metrics.getMeter(METER_NAME).createCounter(name, options)
      this.counters.set(name, counter)
    }
    return counter
  }

Then we just use optional chaining to do nothing if undefined:

  const counter = this.getOrCreateCounter(name)
  counter.?add(...)

But again, this has the problem of doing all the work in incrementCounterWithTransactionAmount for example only to not actually do anything at the end. We'd also need to remember to apply the same pattern if we added additional metrics beyond counter/histograms.

  1. Just add an early return to recordHistogram/incrementCounter if telemetry is disabled. But would the person adding the next similar method remember to do the same? One reason why I think have it at the highest level (as it is in createTelemetryService is good).

Copy link
Contributor

@mkurapov mkurapov Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I don't think its possible - we arent registering metric/trace providers on service init

I see what you mean, what happens to the existing metrics/counters/histogramts when we keep the original/existing telemetry service, but with enableTelemetry disabled?

But again, this has the problem of doing all the work in incrementCounterWithTransactionAmount for example only to not actually do anything at the end

I mean, it might not be a terrible thing, I don't think it really does much when there are no providers configured

Copy link
Contributor Author

@BlairCurrey BlairCurrey Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean, what happens to the existing metrics/counters/histogramts when we keep the original/existing telemetry service, but with enableTelemetry disabled?

I tested this after writing these comments and it's a no-op since src/telemetry/index.ts wont register the meter. So no metrics collected, same as this. We can just go that direction if we're OK with executing the real telemetry service methods where they're called. Honestly this shouldnt be a problem except we are doing more work than simply passing the values into the metric counter/histrogram in the convert methods. Potential network call and applying privacy.

Maybe its not the biggest deal, and presumably new methods shouldn't do any more than this...

I don't think it really does much when there are no providers configured

telemetry meter providers? In src/telemetry/index? I'm not seeing how thats a factor. I think it runs the whole method with rate conversion, privacy, etc. and but never gets kicked up to prometheus because the meter isnt registered.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm with you, having a No-op class is clearer, and we don't end up having any side effects (even if they don't really do anything).

@BlairCurrey BlairCurrey marked this pull request as ready for review September 19, 2024 18:50
mkurapov
mkurapov previously approved these changes Sep 24, 2024
Comment on lines +23 to +25
telemetry:
logs:
level: warn
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do these do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removes a lot of noise from the collector logs. watching the container logs with telemetry enabled is unreadable without it.

Comment on lines 48 to 55
export function createTelemetryService(
deps: TelemetryServiceDependencies
): TelemetryService {
if (!deps.enableTelemetry) {
return new NoopTelemetryServiceImpl(deps)
}
return new TelemetryServiceImpl(deps)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm with you, having a No-op class is clearer, and we don't end up having any side effects (even if they don't really do anything).

Comment on lines 51 to 53
if (!deps.enableTelemetry) {
return new NoopTelemetryServiceImpl(deps)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I prefer having this if (!deps.enableTelemetry) check in src/index.ts instead of here, since it would seem clearer to me to understand how each class/dependency gets registered (kind of like tigerbeetle). Also wouldn't need to pass in enableTelemetry: boolean. NABD, just an option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that makes sense - good to have the switch as high level as possible I think.

Changed to return the noop or real one based on config. enableTelemetry in src/index.ts. In doing so I realized we don't need to pass in all the deps to the noop one so I removed those as well.

@BlairCurrey BlairCurrey merged commit 98ceece into main Oct 1, 2024
30 of 42 checks passed
@BlairCurrey BlairCurrey deleted the bc/2980/refactor-telemetry-optionality branch October 1, 2024 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg: backend Changes in the backend package. type: source Changes business logic type: tests Testing related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make telemetryService defined
2 participants